Feature analysis for automatic speechreading

− Audio-Visual Automatic Speech Recognition systems use visual information to enhance ASR systems in clean and noisy environments. This paper compares of a number of different visual feature extraction methods. When performing visual speech recognition the visual feature vector requires a base level of detail for optimum recognition. Geometric feature extraction provides lower recognition than pixel based methods due to the loss of characteristic speech information such as f-tuck, protrusion etc. Downsampling of images reduces visual recognition scores due to the loss of detail in the images. Also, the role of dynamic features was investigated for improved recognition. It was observed that the use of static features only, provided higher recognition scores than with a feature vector of the same length containing both static and dynamic features. These results illustrate the need for a base level of detail in the feature vector for improved visual recognition scores.